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Executive  Summary 


The  Institute  for  Defense  Analyses  (IDA)  was  tasked  by  the  Office  of  the  Deputy 
Assistant  Secretary  of  Defense  for  Systems  Engineering  (ODASD(SE))  Director,  Mission 
Assurance  to  conduct  research  on  developing  high  level  tools  for  use  by  ODASD(SE). 
The  target  timeframe  within  the  acquisition  process  for  using  these  tools  is  pre-Milestone 
A  and  pre-Milestone  B.  The  objectives  of  IDA’s  task  were  to  evaluate  whether  the 
sponsor’s  research  goals  are  feasible  and  to  provide  a  project  plan  for  addressing  those 
research  goals. 

ODASD(SE)’s  first  research  objective  is  to  understand  the  extent  to  which  systems 
engineering  (SE)  is  being  under-resourced  (funded)  and  how  initial  under-resourcing  of 
systems  engineering  and  program-specific  factors  (e.g.,  percent  software)  is  related  to 
program  outcomes  (i.e.,  additional  cost  and  schedule  growth).  IDA  detennined  that 
developing  Defense  Acquisition  Board  (DAB)-level  systems  engineering  resourcing  rules 
for  DoD  weapon  systems  is  feasible. 

IDA  arrived  at  this  conclusion  based  on  research  undertaken  by  RAND  and  an 
evaluation  of  the  Contractor  Cost  Data  Report  (CCDR)  library  and  IDA’s  cost  growth 
data  set.  The  DAB-level  resourcing  rules  envisioned  in  this  research  would  allow 
ODASD(SE)  to  estimate  the  SE  “should  cost”  for  a  specific  weapon  system. 
Additionally,  the  resourcing  rules  would  outline  the  program  cost  penalty  for  under¬ 
resourcing  SE  (e.g.,  5  percent  reduction  in  SE  funding  would  result  in  a  10  percent 
increase  in  program  costs). 

ODASD(SE)’s  second  set  of  research  objectives  are  to  understand: 

•  if  physics-based  analysis  of  cost  and  capability  trade  spaces  (PACCTS)  is 
feasible, 

•  what  benefits  PACCTS  provides, 

•  the  process  for  conducting  a  PACCTS  project,  and 

•  where  PACCTS  could  play  a  role  within  the  DoD  acquisition  process. 

The  purpose  of  the  physics-based  analysis  is  to  link  capabilities  (e.g.,  key 
perfonnance  parameters)  to  costs  in  order  to  create  a  map,  informed  by  physics,  that 
graphically  shows  the  relationship  between  desired  capabilities  and  the  required  costs. 
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In  the  course  of  this  task  IDA  detennined  that: 


•  PACCTS  is  feasible,  as  demonstrated  by  IDA  Document  D-3744,  “Performance 
Trades  for  Joint  Light  Tactical  Vehicle.”1 

•  Physics-based  analysis  can  expose  decision  makers  to  the  full  range  of  options 
(i.e.,  cost  and  capabilities)  in  a  simple  and  communicable  manner. 

•  Developing  PACCTS  requires  linking  the  cost  and  capabilities  to  physical 
design  features,  which  in  turn  are  governed  by  physics  and  engineering 
principles,  thereby  implicitly  creating  a  link  between  cost  and  capabilities. 

•  ODASD(SE)  has  at  least  three  opportunities  prior  to  Milestone  B  to  use 
PACCTS  to  inform  the  acquisition  process.  The  most  fruitful  and  highest 
leveraged  opportunity  requires  engaging  with  the  initial  capabilities  document 
(ICD)  stakeholders  to  ensure  that  the  desired  capabilities  are  mutually 
compatible. 

Potentially,  PACCTS’s  biggest  benefit  is  that  it  can  map  out  the  feasible  capability 
and  cost  space  for  a  weapon  system.  In  contrast,  Analysis  of  Alternatives  (AoA) 
generally  considers  several  design  points  and  not  the  entire  cost-capability  space. 
Conceptually,  PACCTS  involves  mapping  capabilities  to  physical  design  features  or 
parameters  through  the  use  of  physics  and  engineering  relationships.  We  then  cost  these 
physical  features  or  parameters  (e.g.,  inlet  area  for  an  engine  or  diameter  of  helicopter 
blades).  Since  we  can  link  cost  and  capabilities  to  physical  design  features,  we  can 
compute  the  cost  and  capability  trade  space. 

ODASD(SE)’s  best  opportunity  for  improving  the  acquisition  process,  with 
PACCTS,  occurs  during  the  ICD  formulation  stage.  During  this  phase  ODASD(SE),  with 
the  cooperation  of  the  ICD  stakeholders,  has  the  potential  to  help  identify  which 
requirements  are  compatible  or  incompatible  and  how  they  will  drive  costs  for  major 
acquisition  programs.  Additional  opportunities  to  engage  occur  during  the  AoA  and  after 
Milestone  A,  as  outlined  in  DTM  10-017“  and  an  Office  of  the  Secretary  of  Defense 
(OSD)  memo  entitled  “Preparation  for  Defense  Acquisition  Board  (DAB)  Meetings, 
DAB  Readiness  Meetings  (DRM),  and  DAB  Planning  Meetings  (DPM),”3  respectively. 


David  R.  Gillingham  et  al.,  “Performance  Trades  for  Joint  Light  Tactical  Vehicle” 
(Unclassified//FOUO),  IDA  Document  D-3744  (Alexandria,  VA:  Institute  for  Defense  Analyses,  June 
2009). 

Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics,  Directive  Type  Memorandum 
(DTM)  10-017,  “Development  Planning  to  Inform  Material  Development  Decision  (MDD)  Reviews 
and  Support  Analyses  of  Alternatives  (AoA),”  September  13,  2010. 

Principal  Deputy  Under  Secretary  of  Defense  (OSD),  “Preparation  for  Defense  Acquisition  Board 
(DAB)  Meetings,  DAB  Readiness  Meetings  (DRM),  and  DAB  Planning  Meetings  (DPM),” 
Memorandum,  April  23,  2010. 


IV 


Contents 


Title  Page . 1 

Task  Overview . 2 

Outline  -  Literature  Survey  Results . 3 

Literature  Survey  Objectives . 4 

Findings  Reported  in  SE  Literature . 5 

Caveats  of  Literature  Survey . 7 

Outline  -  Systems  Engineering . 8 

Project  Questions  and  Expected  Results . 9 

Question  1 :  Approach  and  Notional  Results . 10 

Question  2:  Approach  and  Notional  Results . 1 1 

Question  3:  Approach  and  Notional  Results . 12 

Project  Proposal:  Required  Effort  and  Considerations . 13 

Outline  -  PACCTS . 14 

Project  Objective . 15 

What  are  Physics-Based  Analyses  of  Cost  and  Capability  Trade 

Spaces  (PACCTS)? . 16 

Helicopter  Platform . 18 

Physics-Based  Modeling  Process . 20 

JLTV  (1  of  3):  Current  State  of  Program . 22 

JLTV  (2  of  3):  Results  of  Anny  Attribute  Balancing  Analysis . 23 

JLTV  (3  of  3):  Results  of  IDA  JLTV  Analysis . 24 

DoD  Acquisition  Process  Flow:  Places  ODASD(SE)  Can  Play  a  Role  Pre-MS  B . 25 

Physics-Based  Analysis  of  Cost  and  Capability  Trade  Space  Proposal . 26 

Outline  -  Concluding  Remarks . 27 

Concluding  Remarks . 28 

Backup  for  Literature  Survey . 29 

Select  Cost  Model  Cost  Drivers . 30 

INCOSE  Categories  of  SE  Leading  Indicators . 31 

Some  INCOSE  SE  Metrics  of  Leading  Indicators . 32 

Some  Key  NDIA/SEI  SE  Processes . 33 

References . A-l 

Abbreviations . B-l 


v 


Institute  for  Defense  Analyses 

4850  Mark  Center  Drive  -  Alexandria,  Virginia  22311-1882 


Research  on  Development  and 
Application  of  Tools  to  Assist  the  OSD 
Office  of  Systems  Engineering 


AU-7-3307 
June  16,  2011 


l 


I  DA 


Task  Overview 


■  Sponsor:  Director,  Mission  Assurance 

■  Funding:  $  100,000 

■  Summary  of  Statement  of  Work 

a.  Review  and  summarize  key  findings  of  literature  on  the  proper  sizing  of  systems 
engineering  (SE) 

b.  Make  recommendations  on  the  feasibility,  difficulty,  and  likely  content  of  developing 
a  SE  sizing  model 

c.  Explore  the  feasibility  of  developing  physics-based  models  for  cost-performance 
trades  and  how  they  can  be  used  within  the  acquisition  process 

d.  Develop  proposals  for  creating  and  testing  a  SE  sizing  model  and  conducting 
physics-based  trade  studies 

■  Sponsor  asked  for  a  proposal  on  conducting  a  physics-based  assessment 
of  the  Ground  Combat  Vehicle  (not  included  in  this  briefing) 


This  task  is  sponsored  by  the  Director,  Mission  Assurance  in  the  Office  of  the 
Deputy  Assistant  Secretary  of  Defense  (Acquisition,  Technology  and  Logistics)  for 
Systems  Engineering  (ODASD(SE)).  The  goals  of  this  task  are  to  review  and  summarize 
the  key  findings  of  literature  on  the  proper  sizing  of  systems  engineering,  to  propose  a 
project  plan  for  developing  a  systems  engineering  resourcing  model,  and  to  develop  a 
plan  on  how  to  implement  and  use  physics-based  analysis  of  cost  and  capability  trade 
spaces  (PACCTS)  in  the  DoD  acquisition  process.  The  project  team  consisted  of 
members  from  IDA’s  Cost  Analysis  and  Research  Division  (CARD)  and  Science  and 
Technology  Division  (STD).  During  the  six-month  project,  IDA  was  also  asked  to 
provide  a  plan  for  implementing  PACCTS  for  the  Ground  Combat  Vehicle  (GCV) 
program.  The  results  of  the  GCV  project  plan  were  provided  to  the  sponsor  informally 
and  are  not  included  in  this  report. 
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Outline 


1  DA 

■  Literature  Survey  Results 

■  Systems  Engineering  Sizing 

*  Project  Objective,  Approach,  and  Notional 
Results 

■  Project  Proposal 

■  Physics-based  Analyses  of  Cost  and 
Capability  Trade  Space 

*  Project  Objective,  Approach,  and  Notional 
Results 

■  Project  Proposal 

■  Concluding  Remarks 


2 


This  document  is  divided  into  four  parts.  The  results  of  the  literature  survey  outline 
the  current  state  of  research  on  the  proper  resourcing  of  SE,  programmatic  factors  that 
drive  systems  engineering  costs,  and  system  engineering  metrics  that  are  perceived  to  be 
leading  indicators  of  program  outcomes.  The  systems  engineering  sizing  portion  of  the 
document  focuses  on  the  analytical  questions  that  can  be  addressed,  the  expected  results, 
and  the  project  plan.  The  PACCTS  section  of  the  document  focuses  on  the  benefits  of 
conducting  physics-based  analysis,  an  overview  of  how  such  an  analysis  could  be 
conducted,  a  real  world  example  that  demonstrates  the  benefits  of  physics-based  analysis, 
and,  finally,  possible  places  for  ODASD(SE)  to  engage  in  the  acquisition  process.  The 
concluding  remarks  summarize  the  major  points  of  the  document. 
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1  lJy^ _ Literature  Survey  Objectives 

■  Review  and  summarize  key  findings  of 
literature  on  the  proper  sizing  of  SE 

■  Identify  some  SE-related  leading 
indicators  (metrics  and  processes)  of 
program  success  or  failure 


IDA 


Findings  Reported  in  SE  Literature 


■  These  results  are  reported  in  SE  literature  and  are  not 
IDA  conclusions 

■  Proper  sizing  of  SE  results 

■  The  optimal  SE  effort,  to  minimize  cost  growth,  appears  to  be 
approximately  15-20%  of  actual  RDT&E  effort 

■  A  wide  variety  of  quantitative  and  qualitative  factors  are  used 
in  SE  sizing  estimates  (e.g.,  Program  Cost/Duration  and  Type 
of  Item) 

■  Leading  indicators  results 

■  SE  Leading  Indicators  Guide  (v2.0)  identified  18  categories 
and  many  metrics  for  evaluating  SE  (e.g.,  Requirements 
trends) 

■  Several  key  SE  processes  that  correlate  with  program 
success  (e.g.,  Trade  Studies)  are  reflected  in  Capability 
Maturity  Model  Integration  model 

■  See  backup  for  additional  material 
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The  results  of  the  literature  survey  represent  views  that  IDA  found  in  literature  and 
do  not  necessarily  reflect  IDA’s  conclusions.  When  evaluating  the  issue  of  how  to 
properly  size  SE,  in  one  study,  Dr.  Honour  attempted  to  correlate  program  outcomes  (cost 
and  schedule  growth)  with  SE  effort;  other  studies  evaluated  specific  characteristics  of 
the  program.  Dr.  Honour’s  research  indicates  that,  on  average,  cost  and  schedule  growth 
is  minimized  when  SE  effort  is  approximately  15  to  20  percent  of  final  total  program 
development  cost.  There  were  a  variety  of  references  that  identified  parameters  that 
affected  the  way  programs  sized  their  SE  efforts.  The  two  most  useful  references  were  the 
National  Defense  Industrial  Association  (NDIA)/  Software  Engineering  Institute  (SEI) 
study  and  the  Constructive  Systems  Engineering  Cost  Model  (COSYSMO)  dissertation. 
Examples  of  such  parameters  included  Program  Total  Cost,  Program  Duration,  and  Type 
of  Item.  Additional  parameters  and  references  can  be  found  in  the  backup  and 
bibliography,  respectively. 

Additionally,  IDA  found  numerous  reports  that  identified  possible  leading  indicators 
of  program  health.  For  example,  the  SE  Leading  Indicators  Guide  identified  18  broad 
categories,  with  each  category  containing  numerous  recommended  individual  metrics.  As 
an  example,  the  requirements  trends  category  included  such  metrics  as  requirement 
growth  and  requirements  stability.  A  more  complete  list  of  these  categories  is  provided  in 
the  backup  slides,  as  are  additional  references  for  leading  indicators.  One  particular 
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leading  indicator,  addressed  in  the  joint  NDIA/SEI  study,  identified  a  number  of  key  SE 
processes  that  appeared  to  be  correlated  with  program  success. 
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LLLi 1 _ Caveats  of  Literature  Survey 

■  Data  was  primarily  voluntary  and  survey- 
based  (self-selection  bias) 

■  Dataset  included  a  mix  of  DoD,  NASA, 
and  commercial  projects 

■  Very  few  projects  with  SE  effort  >  10% 

■  Limited  confidence  on  optimal  SE  effort 
due  to  data  limitations 


While  the  literature  survey  indicated  possible  avenues  for  continued  exploration, 
there  are  several  significant  caveats  that  must  be  considered  before  applying  the  results  to 
DoD.  The  data  used  for  the  vast  majority  of  research  (e.g.,  Dr.  Honour’s,  the  NDIA/SEI 
study,  COSYSMO,  and  the  Leading  Indicators  Guide)  were  based  on  information  that 
was  provided  voluntarily,  introducing  the  possibility  of  a  self-selection  bias.  Additionally, 
a  large  portion  of  the  data  from  these  four  sources  were  based  on  survey  responses.  While 
some  of  these  data  came  from  DoD  programs,  the  surveys  also  included  data  from 
commercial  sources,  NASA,  and  other  government  agencies.  This  is  explicitly  the  case 
for  Dr.  Honour’s  work  and  the  NDIA/SEI  work.  Furthermore,  due  to  the  need  to  protect 
proprietary  data,  we  do  not  know  which  specific  programs  were  considered.  However, 
given  the  cost  distributions  on  the  programs,  it  is  unlikely  that  many  of  them  were  Major 
Defense  Acquisition  Programs  (MDAPs).  In  looking  at  Dr.  Honour’s  work  on  the  proper 
resourcing  of  SE  effort,  there  were  very  few  projects  with  SE  effort  greater  than  10 
percent  and  the  data  exhibited  a  significant  amount  of  scatter.  Furthermore,  the 
information  was  not  binned  according  to  like  programs,  thereby  limiting  the  applicability 
to  specific  DoD  programs. 
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Outline 


I  DA 

^Literature  Survey  Results 

■  Systems  Engineering  Sizing 

■  Project  Objective,  Approach,  and  Notional 
Results 

■  Project  Proposal 

■  Physics-based  Analyses  of  Cost  and 
Capability  Trade  Space 

■  Project  Objective,  Approach,  and  Notional 
Results 

■  Project  Proposal 

■  Concluding  Remarks 
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In  the  next  section  we  will  discuss  IDA’s  proposal  for  conducting  research  on  how 
to  properly  resource  SE.  Slides  7  through  11  cover  the  analytical  questions  that  can  be 
addressed  and  the  long  term  goals  of  the  research  project,  and  present  a  plan  for 
empirically  addressing  the  analytical  questions. 
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Project  Questions  and  Expected  Results 


Sponsor 

Questions 

Is  SE  under-resourced? 

What  metrics  and  programmatic  factors  drive  SE  costs? 

Analytical 

Questions 

Is  there  a  correlation  between  program  performance  (cost  and  schedule)  and: 

planned  SE  funding 

contractor  reported  SE  metrics 

programmatic  factors  (e.g., 
percentage  software) 

Correlation  between  SE 
funding  and  program 
performance 

Identification  of  key  metrics  and  their 
correlation  to  program  performance 

Identification  of  key  program  factors 
(characteristics)  and  their  correlation 
to  program  performance 

Top-level  cross  checks  and 
rules  govering  SE  funding  / 
penalties  for  underallocation 

Long-term  sponsor  objective:  Cost  model  that  determines 
Systems  Engineering  funding  for  well-behaved  programs 


The  sponsor  of  this  task  is  primarily  interested  in  three  objectives:  understanding 
whether  SE  is  under-resourced  by  MDAPs,  identifying  metrics  and  programmatic  factors 
that  drive  SE  costs,  and  developing  an  SE  “should  cost”  model.  It  may  be  difficult,  if  not 
impossible,  to  address  the  sponsor’s  questions  directly.  Instead,  IDA  proposes  to 
empirically  address  the  three  analytical  questions  in  Slide  7  above.  These  questions  act  as 
proxies  for  the  sponsor’s  questions;  however,  the  responses  to  the  analytical  questions 
can  be  empirically  derived.  Furthermore,  addressing  the  three  analytical  questions  should 
allow  IDA  to  begin  addressing  ODASD(SE)’s  long-term  objective.  The  next  three  slides 
will  present  IDA’s  project  plan  for  addressing  the  three  analytical  questions. 
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I  DA 


Question  1:  Approach  and  Notional 

Results 


How  does  initial  contractor  SE  budgeting  affect  program  outcomes? 


Distribution  of  Development  Cost 
Growth  From  MS  II  Baseline  For  151 
Major  Defense  Acqusition  Progra^l^ 

MS 


1.0  0  0,0  □□ 


-  .A  n  .O  \0  vO  sO  «p  .0  ,0  .O  V' 


V. 


%  Cost  Growth 


+ 

“V 


Development  Cost  Growth  as  a  Function  of  SE 
Resourcing  (by  Commodity,  Service,  etc.) 


■  Contractor  SE  accounts  for  B%  of  total 
Development  for  "mature"  MDAPs 

■  Programs  with  less  than  25%  cost  growth 
had  initial  SE  resourced  at  C% 

■  Programs  with  25%  or  more  cost  growth  had 
initial  SE  resourced  at  D% 

■  Programs  that  under-resource  SE  typically 
pay  an  X%  penalty  in  additional  SE  and  Y% 
in  penalty  in  development  cost  growth 


This  slide  depicts  the  notional  integration  of  MDAP  cost  growth  infonnation 
obtained  from  Selected  Acquisition  Reports  with  SE  information  from  Contractor  Cost 
Data  Reports  (CCDR).  The  goal  of  this  approach  will  be  to  evaluate  the  growth  (e.g., 
RDT&E,  procurement)  each  MDAP  had  as  a  function  of  initial  SE  as  planned  at  the 
beginning  of  the  program.  Cost  growth  will  also  be  compared  to  the  total  final  amount  of 
SE  funding  (at  the  end  of  the  program  or  most  recently  reported  expenditure).  From  this 
study  design,  we  will  calculate  an  average  RDT&E  funding  budgeted  for  SE  by  type  of 
MDAP  (e.g.,  size,  commodity  type,  and  Service).  More  importantly,  we  will  evaluate 
how  initial  conditions  of  SE  funding  affected  total  cost  growth  for  MDAPs  of  varying 
size,  commodity  type,  and  Service.  This  analysis  should  enable  us  to  empirically  identify 
the  correlation  between  under-resourcing  initial  SE  and  cost  growth  (e.g.,  development, 
procurement,  SE).  It  may  also  highlight  “knees”  in  the  curve  that  could  indicate  how 
much  SE  funding  is  required  on  average  to  minimize  cost  growth  in  MDAPs. 
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IDA 


Question  2:  Approach  and  Notional 

Results 


■  Objective: 

■  Identify  to  what  extent  SE  metrics  (or  lack  thereof)  affect 
MDAP  outcomes 

■  Approach: 

■  Collect  and  identify  which  SE  metrics  are  used  in  MDAPs 

■  Merge  SE  related  metrics  into  dataset 

■  Correlate  the  use  of  SE  metrics  (or  lack  thereof)  with 
program  outcomes 

■  Output: 

■  Increased  understanding  of  the  extent  to  which  SE  metrics 
are  reported  and  used  to  manage  programs 

■  Estimate  effect  of  metrics  (e.g.,  Not  using  SE  metrics 
results  in  X%  cost  growth  on  average) 

■  Develop  heuristics  (e.g.,  SE  metrics  YY  and  ZZ  are  the 
most  correlated  with  low  cost  growth  programs) 
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In  this  second  study  we  will  identify  the  SE-related  metrics  (and  practices)  (e.g., 
requirements  trends  and  technical  measurement  trends)  actually  used  by  programs  to 
manage  risk  and  correlate  the  use  of  these  metrics  with  program  success  by  building  on 
the  information  obtained  in  addressing  Question  1.  This  second  study  goes  beyond 
current  research  on  SE-related  leading  indicators  in  that  it  will  rely  on  evidence  from  all 
DoD  programs  that  comply  with  reporting  policies  instead  of  just  programs  that  want  to 
report. 
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Question  3:  Approach  and  Notional 

Results 


■  Objective: 

■  Understand  how  program-specific  characteristics  will  affect 
SE  funding  requirements  and  program  performance 

■  Approach: 

■  Identify,  collect,  and  integrate  top-level  programmatic 
factors  (e.g.,  percentage  software,  number  of  new 
technologies)  into  dataset 

■  Determine  how  SE  effort  and  program  outcomes  depend 
on  key  programmatic  factors 

■  Output: 

■  Identification  of  which  factors  are  correlated  with  SE  costs 
and  program  performance 

■  Understand  the  magnitude  of  specific  programmatic 
properties  on  SE  resourcing  requirements  and  program 
performance 


During  the  third  phase,  we  will  determine  what  factors  appear  to  have  the  greatest 
impact  on  the  amount  of  SE  resources  required  to  successfully  execute  a  program.  To  this 
end,  we  will  select  programmatic  factors  for  consideration  (guided  by  results  of  literature 
search)  and  look  for  cross-correlations  between  the  selected  factor,  systems  engineering 
effort,  and  program  success.  We  will  attempt  to  derive  rules  that  allow  for  the  estimation 
of  SE  resources  required,  based  on  program-specific  characteristics.  This  third  study  goes 
beyond  current  research  in  that  it  will: 

•  Focus  on  DoD  MDAPs,  which  is  a  limitation  of  the  NDIA/SEI  study  and 
COSYSMO. 

•  Expand  the  scope  of  the  RAND  study  by  considering  all  DoD  MDAPs  and  not 
just  Air  Force  aircraft  and  guided  weapon  programs. 

•  Be  based  on  government-mandated  contractor  cost  data,  which  addresses  a 
serious  limitation  of  the  NDIA/SEI  study  and  COSYSMO. 
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IDA 


Project  Proposal:  Required  Effort  and 

Considerations 


■  Execute  under  a  single  task  order  with  three  sequential  phases 

■  Phase  1 :  SE  budgeting  vs.  MDAP  outcomes 

■  OSD  contractor  cost  data  web  archive  contains  relevant  information, 
but  limited  to  <40  programs 

■  Data  collection  >4  staff  months 

■  Provide  results  of  analysis 

■  Phases  2  and  3:  SE  metrics  and  program  factors  vs.  MDAP 
outcomes 

■  No  central  web  archive  of  SE,  technical,  and  programmatic 
information 

■  Will  have  to  collect  and  evaluate  information  from  SE  and  program 
offices 

■  Provide  results  of  analysis  as  each  phase  is  completed 

■  Can  look  at  alternative  data  sources  but  it  will  require  additional 
resources 

■  Research  Team:  ~3-4  research  staff 


IDA  proposes  that  the  project  be  conducted  under  a  single  task  order  with  individual 
statements  of  work  to  cover  each  analytical  question.  This  approach  should  allow  IDA  to 
minimize  the  time  required  to  collect  the  required  information.  Some  issues  to  consider 
include  the  fact  that  the  quality  of  the  cost  data  is  uncertain  and  the  data  collection  effort 
will  probably  dominate  the  first  four  months  of  the  task.  The  research  team  will  be  made 
up  of  three  or  four  research  staff  members,  who  will  undertake  management  and  analysis 
roles.  Several  additional  researchers  will  be  used  to  front  load  the  information  gathering 
phase,  thereby  reducing  the  calendar  time  it  takes  to  begin  the  analysis. 
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In  the  section  on  PACCTS,  we  will  define  PACCTS,  provide  an  overview  on 
implementing  and  conducting  it,  and  give  a  real  world  example  of  its  benefits. 
Additionally,  we  will  discuss  how  PACCTS  can  be  used  within  the  DoD  acquisition 
process  and  provide  a  framework  for  conducting  weapon  system-specific  analysis. 
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LMd i _ Project  Objective 

■  In  a  transparent  and  communicable  manner, 
display  the  cost  and  capability  space  of  a 
weapon  system  using  pre-Milestone  A-like 
information 

■  Identify  trade  space,  not  just  design  points 

■  Give  decision  makers  the  full  range  of  options 

■  Identify  cost  drivers  and  how  they  are  related 
to  system  level  technologies 

■  Get  at  why  and  what  makes  it  expensive 

Want  to  confidently  state:  “Capability  X  will  cost  Y 

because  of  Z” 


The  top-level  purpose  of  PACCTS  is  to  be  able  to  clearly  and  concisely 
communicate  to  a  decision  maker  the  range  of  capabilities  that  can  be  supported  by  a 
system,  the  associated  costs  of  the  different  capability  levels,  and  the  driving  reason  for 
the  associated  costs.  The  ability  to  effectively  communicate  is  a  significant  part  of  this 
project  and  was  not  treated  as  a  secondary  objective.  Combining  multidimensional 
capability  and  technical  data  with  cost  can  be  overwhelming.  With  this  in  mind  IDA 
focused  a  significant  part  of  its  effort  to  outlining  a  process  that  lays  out  how  design 
features,  system  capabilities,  and  cost  can  be  combined  into  a  visual  chart  that  captures 
the  relevant  information.  In  practice  we  expect  that  the  final  visual  product  will  vary 
depending  on  the  weapon  system  and  the  factors  that  influence  the  decision  metric  (e.g., 
time,  cost,  risk,  or  quantity). 
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IDA 


What  are  Physics-based  Analyses  of  Cost 
and  Capability  Trade  Spaces  (PACCTS)? 


■  Use  the  physically  achievable  design  space 
to  map  out  the  feasible  capability  trade  space 

■  Estimate  costs  based  on  physical  design 
parameters 

■  Proof  of  Principle: 

■  Performance  Trades  for  the  Joint  Light  Tactical 
Vehicle  (JLTV)  (IDA  Document  D-3744) 

■  Military  Aircraft  Development  Cost  Volume  1 
(IDA-M-253-VOL-1-REV) 


r  'i 

Modeling  of 
subsystem: 
Helicopter 

L _ _ _ / 

r  'i 

Overview  of 
PACCTS  process: 
Tactical  Missile 

v - / 

r  'i 

Practical  example: 
Joint  Light  Tactical 
Vehicle 

v - / 

r  ^ 

PACCTS  role  in  the 
acquisition  process 

In  previous  parts  of  this  report,  PACCTS  has  been  discussed  broadly  but  it  has  not 
yet  been  clearly  defined.  The  essence  of  PACCTS  is  to  use  the  physically  achievable 
design  space  to  map  out  the  feasible  capability  trade  space.  Additionally,  PACCTS  looks 
to  estimate  the  costs  based  on  physical  design  parameters. 

Essentially  PACCTS  is  designed  to  combine  three  key  features  into  a  single  process, 
thereby  directly  linking  costs  to  capabilities.  One  of  the  key  features  of  PACCTS  is  that  it 
uses  physics  and  engineering  principles  as  drivers  of  capabilities  and  cost,  thereby  linking 
the  desired  macro  level  programmatic  properties  with  the  micro  level  design  features  that 
will  drive  designer  decisions.  Additionally,  the  analysis  is  conducted  over  the  feasible 
design  space,  thereby  presenting  the  full  range  of  options  as  opposed  to  isolated  design 
points.  Finally,  it  presents  the  benefits  and  consequences  of  the  different  options  in  a 
clear,  transparent  manner  without  making  value  judgments. 

This  section  of  the  report  is  divided  into  four  parts.  In  the  first  part,  the  report  will 
discuss  how  to  analytically  implement  PACCTS  (e.g.,  by  discussing  an  equation  for  a 
system).  Next,  we  provide  an  overview  of  how  PACCTS  can  be  conducted  and 
implemented  end  to  end.  Thirdly,  we  discuss  the  Joint  Light  Tactical  Vehicle  (JLTV)  and 
how  it  could  have  benefited  from  physics-based  analysis.  Specifically,  we  discuss  the 
current  state  of  the  JLTV  program,  the  Analysis  of  Alternatives  (AoA)-like  process  that 
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led  up  to  the  current  conundrum,  and  the  IDA  physics-based  analysis  that  highlights 
several  key  issues  that  the  program  is  currently  attempting  to  address.  Finally,  we  discuss 
several  possible  places  for  PACCTS  to  be  used  by  ODASD(SE)  to  inform  and  improve 
the  acquisition  process. 
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Helicopter  Platform 


15 


This  example  discusses  how  PACCTS  can  be  implemented  at  the  system  level. 
Specifically,  this  example  is  for  a  helicopter  platform  in  hover.  The  end  goal  of  PACCTS 
is  to  be  able  to  link  cost  and  capabilities  to  key  physical  design  parameters  of  a  weapon 
system  like  the  Apache,  as  shown  in  the  upper  right  hand  comer  of  Slide  15.  While  being 
able  to  model  a  fully  functioning  weapon  system  like  the  Apache  is  the  goal,  it  is  not 
where  the  analysis  begins.  The  analysis  begins  with  a  simple  equation,  to  which  we 
iteratively  add  complexity  and  detail  until  we  have  a  model  that  represents  the  weapon 
system.  We  begin  by  modeling  a  simple  and  idealized  version  of  the  helicopter  akin  to  the 
“whirligig”  toy  (upper  left  hand  corner).  The  equation  below  the  figures  is  an  idealized 
helicopter  equation  and  provides  a  relationship  between  the  power,  blade  radius,  air 
density,  and  weight  of  the  vehicle.  This  represents  the  starting  point  of  the  analysis,  from 
which  we  will  discuss  how  it  can  be  refined  to  incorporate  additional  information.  We  use 
this  equation  as  an  explanatory  tool;  in  practice,  we  will  have  to  solve  multiple 
simultaneous  equations. 

The  first  stage  in  the  process  is  to  identify  the  relevant  variables,  the  dependencies 
between  them,  and  how  capabilities  can  be  incorporated  into  the  equations.  Using  the 
ideal  helicopter  equation  as  a  guide,  we  will  discuss  how  the  power  (P),  blade  radius  (R), 
air  density  (p),  and  air  vehicle  weight  (W)  depend  on  various  physical  processes,  the 
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mission,  and  different  technological  limitations.  Additionally,  we  will  discuss  how  they 
are  linked  to  one  another  through  several  underlying  physical  attributes,  which  are  not 
immediately  apparent. 

The  power,  P,  represents  the  useful  energy  per  unit  time  that  is  transferred  into  the 
air.  The  amount  of  power,  P,  that  can  be  delivered  will  depend  on  a  host  of  factors, 
including  mechanical  transmission  efficiencies,  fuel  flow  rates,  allowable  exit 
temperatures,  air  flow  inlet  area,  and  material  properties.  These  variables  encompass  a 
host  of  factors  including,  the  underlying  physics,  the  role  or  mission  of  the  vehicle,  and 
fundamental  material  properties. 

The  blade  radius,  R,  is  constrained  by  the  operational  environment,  the 
aerodynamics,  and  structural  mechanics.  R  may  have  minimum  and  maximum  constraints 
based  on  where  the  vehicle  must  operate  (e.g.,  ship  operations)  and  it  will  be  limited  by 
the  local  speed  of  sound  and  the  torsional  rigidity  of  the  materials  used  in  the  blades. 
Once  again,  we  have  discussed  several  factors  that  will  affect  the  blade  radius  that  are 
linked  to  basic  issues  concerning  the  mission,  physics,  and  fundamental  material 
properties. 

The  air  density,  p,  is  an  atmospheric  property  that  depends  on  the  maximum  altitude 
and  temperature  (i.e.,  detennined  by  mission  and  location). 

Finally,  the  weight,  W,  can  be  built  up  based  on  the  number  of  personnel,  payload 
and  armor  requirements,  and  desired  range.  The  range  will  impact  weight  because  it 
depends  on  the  fuel  tank  size,  fuel  flow  rate,  combustion  temperatures,  air  density,  and 
flight  time.  Once  again,  we  see  a  link  between  capabilities  (e.g.,  range)  and  physics  (e.g., 
fuel  flow  rates  and  combustion  temperature). 

In  the  PACCTS  process  we  would  derive  a  physics-based  model  for  each  subsystem 
and  then  link  them  together  in  order  to  derive  the  system  level  performance.  This  slide 
demonstrates  how  we  can  begin  the  process  by  starting  with  an  idealized  equation  and 
then  expand  it  to  identify  the  relevant  dependencies.  The  next  section  will  place  this 
analysis  in  context  and  walk  through  a  notional  example. 
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Physics-Based  Modeling  Process 
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Process  Overview 
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Slide  16  provides  an  overview  of  the  PACCTS  process  for  an  air-to-air  tactical 
missile.  Before  we  discuss  the  analysis  portion  of  the  process,  it  is  helpful  to  discuss  the 
final  output  or  deliverable.  “The  Cost  and  Capability  (KPPs)  Map,”  on  the  right  hand  side 
of  the  slide,  is  a  notional  figure  that  showcases  the  key  features  that  PACCTS  is  designed 
to  elucidate.  Three  key  aspects  of  the  figure  are:  it  differentiates  between  feasible  design 
space  and  infeasible  design  space,  the  axes  are  linked  to  capabilities  (e.g.,  KPPs),  and  it 
applies  a  cost  to  the  feasible  space. 

In  this  “Map,”  we  have  three  constraints:  platform  constraints,  a  safety  limit,  and  the 
minimum  lethality  required  for  the  mission.  The  internal  volume  and  weight  limit  of  the 
F-22  weapons  bay  would  be  a  type  of  platform  constraint.  The  safety  limit  would 
represent  the  tradeoff  between  explosive  power  of  the  warhead  and  the  minimum  usable 
distance  that  ensures  the  aircraft  is  not  endangered.  The  minimum  lethality  needed  for  the 
mission  would  represent  the  lower  bound  on  lethality  required  in  order  to  fulfill  the 
mission.  This  constraint  does  not  represent  the  desired  level  of  capability  but  simply  the 
absolute  minimum  required  before  the  system  would  be  ineffective  (i.e.,  below  this  limit 
it  would  not  take  down  the  target). 

The  second  aspect  mentioned  above  is  that  the  axes  are  tied  to  system  level 
capabilities,  such  as  range  and  lethality  (e.g.,  probability  of  kill  per  target  area).  We  note 
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that  capabilities  are  system-level  attributes  that  depend  on  the  interaction  of  multiple 
subsystems.  Finally,  a  cost  metric  (i.e.,  life  cycle  costs  or  program  average  unit  cost)  is 
applied  to  the  feasible  space.  In  this  notional  example  we  color  code  the  costs  from  blue 
(low)  to  red  (high). 

The  cost  contours  (same  cost-changing  capabilities)  are  likely  to  follow  immutable 
constraint  boundaries  (e.g.,  weight  or  volume  limits).  Furthermore,  the  costs  sharply 
increase  as  we  approach  an  immutable  constraint,  since  reduced  margin  increases  the 
difficulty  of  the  design  problem.  Also,  the  figure  does  not  prescribe  the  “best  solution;” 
instead  it  describes  how  affordability  will  change  with  the  desired  system-level 
perfonnance,  thereby  letting  the  decision  makers  choose  without  biasing  their  selection. 

So  far  we  have  shown  how  we  would  analyze  a  specific  subsystem  (the  helicopter 
platfonn)  and  we  have  shown  a  chart  (“Cost  and  Capability  Map”)  that  is  a  representative 
output  of  the  PACCTS  process.  Now  we  will  discuss  at  a  high  level  how  we  get  from  the 
weapon  system  concept  to  the  “Cost  and  Capability  Map.”  The  first  step  is  to  break  down 
the  weapon  system  into  its  constituent  subsystems.  In  this  example  of  a  tactical  air-to-air 
missile,  we  list  several  subsystems:  the  seeker,  warhead,  guidance  navigation  and  control, 
battery,  and  rocket  motor.  Once  we  have  the  governing  equations  for  each  subsystem  we 
then  map  them  to  designable  technical  performance  metrics.  These  are  designable 
subsystem-level  metrics  that  will  govern  perfonnance.  For  example,  accuracy  could  be 
measured  as  the  expected  miss  distance  of  the  guidance  system.  Field  of  view  determines 
how  much  the  seeker  can  see.  Finally,  the  designable  technical  performance  metrics 
would  be  combined  to  obtain  the  system  level  perfonnance,  such  as  lethality. 
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The  JLTV  is  a  real  world  example  that  demonstrates  the  potential  value  of  physics- 
based  analysis.  The  JLTV  Category  A  vehicle  is  to  be  a  single  armored  light  wheeled 
vehicle  that  satisfies  both  the  Anny  and  U.S.  Marine  Corps  (USMC).  Currently,  the 
values  assigned  to  the  two  Services’  requirements — namely,  the  Army’s  need  for  force 
protection  and  the  USMC’s  need  for  a  mobile  and  transportable  vehicle — conflict.  While 
RDT&E  is  supposed  to  highlight  and  address  difficult  issues  before  procurement,  in  this 
case  the  conflict  between  force  protection  and  mobility  and  transportability  was  entirely 
predictable  and  highlighted  in  an  IDA  document4  using  pre-Milestone  A-like  information. 


David  R.  Gillingham  et  al.,  “Performance  Trades  for  Joint  Light  Tactical  Vehicle” 
(Unclassified//FOUO),  IDA  Document  D-3744  (Alexandria,  VA:  Institute  for  Defense  Analyses,  June 
2009). 
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JLTV  (2  of  3):  Results  of  Army  Attribute 

Balancing  Analysis 
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■  Army  report  evaluated  six  point  designs  (Category  A 
vehicle) 

■  Constrained  force  protection,  payload,  and  mechanical 
requirements 

■  All  other  KPPs  were  traded  away  (e.g.,  mobility  and 
transportability) 

■  No  consideration  for  affordability 

■  Outcome: 


■  Heavily-armored,  low-transportable  vehicle 

■  Qualitatively  recognized  force  protection  and  weight  linked 


Source:  Joint  Light  Tactical  Vehicle  Attribute  Balancing  Analysis  Final  Report 


The  JLTV  program  began  its  journey  similar  to  most  MDAPs.  It  underwent  an  AoA- 
like  process.  Specifically,  the  JLTV  program  conducted  an  Evaluation  of  Alternatives 
(EoA)  followed  by  an  Attribute  Balancing  Analysis  (ABA).  The  EoA  looked  at  a  broad 
set  of  vehicles  such  as  commercial  solutions,  modifications  to  existing  vehicles,  and  new 
starts.  It  detennined  that  a  new  start  was  the  preferred  option.  The  ABA  determined  what 
attributes  (e.g.,  how  much  force  protection,  payload,  mobility,  etc.)  the  vehicle  should 
have.  Instead  of  fully  considering  the  entire  trade  space,  the  ABA  constrained  the  force 
protection,  payload,  and  mechanical  requirements,  which  limited  its  trades  to  mobility 
and  transportability.  Furthermore,  the  ABA  failed  to  consider  how  the  affordability  would 
be  affected.  Due  to  the  predefined  constraints  on  force  protection,  payload,  and 
mechanical  requirements,  the  results  were  biased  toward  a  heavily  armored  and 
minimally  transportable  vehicle.  Unfortunately,  we  do  not  know  the  reason  for  the  bias 
(e.g.,  institutional,  management  guidance,  faulty  analysis).  While  the  ABA  failed  to  fully 
explore  the  design  space,  it  did  qualitatively  recognize  that  force  protection  and  weight 
are  linked. 
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JLTV  (3  of  3):  Results  of  IDA  JLTV 
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■  Analysis  used  pre-milestone  A-like  information 

■  IDA  evaluated  trade  space  for  JLTV-like  vehicle 

■  Quantified  relationship  between  force  protection  and 
KPPs 


■  Quantified  consequences  of  requirement  choices 
(e.g.,  commonality  and  add-on  armor  kit) 

■  Laid  out  the  possibilities  and  consequences  without 

prescribing  the  answer _ 

Use  of  trade  space  studies  early  in  the  acquisition  process  would 
have  highlighted  key  challenges  and  reduced  development  time 


In  2009,  IDA  was  asked  to  analyze  the  JLTV  to  understand  how  the  different  KPPs 
interacted.  For  this  study,  documented  in  IDA  Document  D-3744,  IDA  used  pre- 
Milestone  A-like  infonnation  and  conducted  a  physics-based  analysis  of  the  JLTV.  In  the 
course  of  the  study,  IDA  quantified  the  tradeoff  between  force  protection  and  mobility 
and  transportability,  as  well  as  other  KPPs.  Specifically,  IDA  quantified  the  tradeoff 
between  increased  force  protection  and  decreased  mobility  and  transportability.  IDA  also 
identified  the  consequences  (in  terms  of  additional  weight)  for  using  a  common  chassis 
and  add-on  armor  kits.  Furthermore,  the  results  were  presented  in  a  manner  that 
highlighted  the  consequences  of  the  different  options  without  prescribing  a  particular 
solution  as  optimal.  This  study  highlights  two  key  issues:  first,  that  physics-based 
analysis  is  feasible  with  pre-Milestone  A-like  information,  and  second,  conducting 
physics-based  analysis  early  on  could  have  highlighted  critical  challenges,  thereby  giving 
managers  sufficient  time  to  address  the  issues  and  avoid  costly  delays  late  in  the  program. 
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I  DA 


DoD  Acquisition  Process  Flow:  Places 
ODASD(SE)  Can  Play  a  Role  Pre-MS  B 


ICD: 

Initial  requirements  (JS) 
AoA  study  guidance: 
Generic  but  usually 
references  ICD  (CAPE) 


AoA:  Trade  space  between 
cost  and  capability  (CAPE) 


•  CDD 

•  PDR  Assessment 

•  Data  Management 

•  Program  Risk 
Assessment 


SE's  1st  Opportunity: 
Ensure  requirements  set 
forth  in  ICD  are  reasonable 
and  self-consistent 
(No  formal  Document) 

SE's  2nd  Opportunity: 
Provide  technical  support  to 
the  AoA  study  guidance  and 
advisory  group 
(DTM  10-017) 

SE's  3rd  Opportunity: 
Program  Risk  Assessment  can 
address  technical  feasibility  and 
possible  cost  implications 
(Kendall  April  23,  2010  Memo) 

Use  analysis  as  early  as  possible:  biggest  bang  for 
buck  comes  from  identifying  and  avoiding  problems 
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IDA  was  able  to  identify  at  least  three  places  that  ODASD(SE)  can  use  physics- 
based  analysis  to  inform  the  acquisition  process.  The  earliest  opportunity  is  to  work  with 
the  Joint  Staff  (JS)  to  ensure  that  the  desired  capabilities  outlined  in  the  initial  capability 
document  (ICD)  correspond  to  an  affordable  and  designable  weapon  system. 
Unfortunately,  IDA  was  not  able  to  locate  any  formal  document  that  requires  the  JS  to 
consult  with  ODASD(SE);  therefore,  this  interaction,  while  beneficial,  would  be 
voluntary.  The  next  opportunity  IDA  was  able  to  identify  is  during  the  material 
development  decision  (MDD).  During  this  phase,  ODASD(SE)  can  serve  in  an  advisory 
capacity  to  Cost  Assessment  and  Program  Evaluation  (CAPE).  The  interaction  between 
CAPE  and  ODASD(SE)  is  outlined  in  DTM  10-017.  Finally,  post  Milestone  A  and  pre 
Milestone  B,  ODASD(SE)  can  engage  in  the  acquisition  process  using  physics-based 
analysis  to  evaluate  program  risk  and  technical  feasibility  and  the  possible  cost 
implications.  The  specific  roles  for  ODASD(SE)  are  documented  in  a  April  23,  2010 
memo  by  the  Principal  Deputy  Under  Secretary  of  Defense  for  Acquisition,  Technology 
and  Logistics.  An  implication  of  the  JLTV  study  is  that  using  physics-based  analysis  early 
in  the  acquisition  process  will  provide  the  most  benefit  by  allowing  DoD  to  identify  and 
avoid  issues  before  they  become  problems. 
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Physics-based  Analysis  of  Cost  and 

LE24 _ Capability  Trade  Space  Proposal 

■  Previously  proposed  Ground  Combat  Vehicle 
(GCV)  task  (if  funded)  should  demonstrate 
utility  of  physics-based  analysis 

■  Draft  a  general  task  order  for  conducting 
physics-based  analysis;  use  amendments  to 
select  and  fund  weapon  system-specific 
analysis 

■  Start  of  analysis  should  begin  ~1  year  prior  to 
a  weapon  system’s  MS  A 

■  Exact  funding  and  schedule  will  depend  on 
weapon  system 


Currently,  we  have  outlined  how  PACCTS  could  be  implemented,  where  it  could  be 
used  within  the  DoD  acquisition  process,  and  its  benefits.  The  resources  required  to 
undertake  PACCTS  will  heavily  depend  on  the  weapon  system.  For  example,  IDA’s  GCV 
proposal  to  ODASD(SE)  built  on  IDA’s  JLTV  study  and,  based  on  the  limited  scope  of 
work,  we  estimated  it  could  be  done  in  approximately  four  months.  We  propose  that  the 
path  forward  incorporate  a  task  order  (IDA  has  one  drafted)  that  describes  at  a  high  level 
the  type  of  analysis  that  will  be  conducted.  This  allows  for  the  use  of  amendments  to  fund 
weapon  system-specific  analysis,  thus  reducing  administrative  delays.  Ideally,  the 
analysis  should  begin  well  before  Milestone  A  so  that  the  results  can  infonn  the  DAB. 
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Outline 


1  DA 

^Literature  Survey  Results 
S Systems  Engineering  Sizing 

^Project  Objective,  Approach,  and  Notional 
Results 

v'  Project  Proposal 

s  Physics-based  Analyses  of  Cost  and 
Capability  Trade  Space  (PACCTS) 

^Project  Objective,  Approach,  and  Notional 
Results 

v'  Project  Proposal 

■  Concluding  Remarks 
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Ly^i _ Concluding  Remarks 

■  Literature  does  not  adequately  quantify  the  value 
of  SE  or  how  to  size  SE  for  DoD  MDAPs 

■  SE  sizing  and  physics-based  analysis  projects  are 
feasible,  as  other  projects  have  conducted  similar 
work  on  a  limited  scale 

■  SE  sizing  project  goals: 

■  Develop  program-specific  heuristics  for  evaluating 
contractor  SE  funding  and  metrics 

■  Identify  a  set  of  top-level  cross  checks  for  SE 
reporting  and  budgeting 

■  Physics-based  analysis  can  identify  key  program 
challenges  and  consequences  early  in  acquisition 

■  Allow  acquisition  process  to  preemptively  address 
issues  before  they  become  problems 
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Institute  for  Defense  Analyses 

4850  Mark  Center  Drive  "Alexandria,  Virginia  22311-1882 


Backup  for  Literature  Survey 
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Ly^i _ Select  Cost  Model  Cost  Drivers 

■  Number  and  difficulty  of  requirements, 
interfaces,  critical  algorithms,  and 
operation  scenarios 

■  Requirements  architecture 

■  Team  experience 

■  Number  and  diversity  of  installations  or 
platforms 

■  Process  maturity  and  team  capability 


A  partial  list  of  SE-related  program  cost  drivers  was  drawn  from  the  papers  by  Dr. 
Valerdi  and  the  NDIA/SEI  Study. 
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I  DA 


INCOSE  Categories  of  SE  Leading 

Indicators 


■  Requirements  Trends 

■  System  Definition  Change  Backlog  Trends 

■  Interface  Trends 

■  Requirements  Validation  Trends 

■  Requirements  Verification  Trends 

■  Work  Product  Approval  Trends 

■  Review  Action  Closure  Trends 

■  Technology  Maturity  Trends 

■  Risk  Exposure  Trends 

■  Risk  Treatment  Trends 

■  Systems  Engineering  Staffing  and  Skills  Trends 

■  Process  Compliance  Trends 

■  Technical  Measurement  Trends 

■  Facility  and  Equipment  Availability 

■  Defect  and  Error  Trends 

■  System  Affordability  Trends 

■  Architecture  Trends 

■  Schedule  and  Cost  Pressure 


26 


This  list  of  Categories  of  SE  Leading  Indicators  was  taken  from  the  INCOSE  2010 
“SE  Leading  Indicators  Guide,  Ver.  2.0.”  Each  Category  includes  a  set  of  metrics  that  are 
recommended  for  use  as  leading  indicators.  An  example  is  provided  in  the  following 
slide. 
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I  DA 


Some  INCOSE  SE  Metrics  of  Leading 

Indicators 


■  Requirements  Trends:  %  Requirements  growth  and  modifications 

■  Interface  Trends:  %  Interface  growth  and  changes 

■  Requirements  Validation  Trends:  %  Requirements  validated 

■  Requirements  Verification  Trends:  %  Requirements  verified 

■  Technology  Maturity  Trends:  Technology  Readiness  Level  for  each 
Critical  Technology  over  time,  Cumulative  Actual  Cost  for  Realization  of 
Technology  Readiness  Levels  (for  a  critical  technology),  Actual  Time  to 
Realization  for  Realization  of  Technology  Readiness  Levels  (for  a  critical 
technology) 

■  Risk  Exposure  Trends:  Projected  risk  exposure  (i.e.,  risk  mitigation  plan) 
vs.  Actual  Risk  Exposure  for  each  risk  of  interest 

■  Risk  Treatment  Trends:  %  of  Risks  that  met  risk  reduction  plans 

■  Systems  Engineering  Staffing  and  Skills  Trends:  Planned  SE 
Effort/Planned  Total  Effort  and  Actual  SE  Effort/Actual  Total  Effort 

■  Technical  Measurement  Trends:  Measured  TPM  value  vs.  Planned  TPM 
value  and  deviation  in  performance 

■  System  Affordability  Trends:  Baseline  Cost/Schedule  (with  confidence) 
and  Planned  Cost/Schedule  (with  confidence) 

■  Schedule  and  Cost  Pressure:  Contract  Cost/Schedule,  Planned 
Cost/Schedule,  and  Actual  Cost/Schedule 

Track  metrics  over  time  to  estimate  performance 


This  chart  indicates  some  of  the  SE-related  metrics  that  may  be  used  as  leading 
indicators  of  program  performance  that  fall  into  each  of  the  Leading  Indicator  categories. 
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Ly£l _ Some  Key  NDIA/SEI  SE  Processes 

■  Requirements  Development  and 
Management 

■  Product  Architecture 

■  Trade  Studies 

■  Technical  Solution 

■  Risk  Management 

■  Verification  and  Validation 


This  slide  indicates  the  six  SE-related  processes  that  the  NDIA/SE  study  found  to  be 
most  highly  correlated  with  program  success. 
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CDD 
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Constructive  Systems  Engineering  Cost  Model 
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Defense  Acquisition  Board 

DAU 

Defense  Acquisition  University 

DoD 

Department  of  Defense 

DPM 

DAB  Planning  Meeting 

DR&E 

Defense  Research  and  Engineering 

DRM 

DAB  Readiness  Meeting 

DSMC 

Defense  Systems  Management  College 

DTM 

Directive-type  Memorandum 

EoA 

Evaluation  of  Alternatives 

FTE 

Full  Time  Equivalent 

GCV 

Ground  Combat  Vehicle 

ICD 

Initial  Capabilities  Document 

IDA 

Institute  for  Defense  Analyses 

IEC 

International  Electrotechnical  Commission 

IEEE 

Institute  of  Electrical  and  Electronics  Engineers 

INCOSE 

International  Council  on  Systems  Engineering 

ISO 

International  Organization  for  Standardization 

JLTV 

Joint  Light  Tactical  Vehicle 

JS 

Joint  Staff 

KPP 

Key  Performance  Parameter 

MDAP 

Major  Defense  Acquisition  Program 

MDD 

Material  Development  Decision 

MS 
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